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Initial Development and Piloting of a Learning-Based, Oassroom Assessment 
and Consultation System: New Perspectives on the Rhetoric of 
Improving Instruction in Higher Education Settings 

The quality of undergraduate teaching and learning environments on college campuses is 
a recent national concrm. The recent call to **ieafQrm teaching as tbt university's primary task"* 
is echoed by die Carnegie Foundation for the Advancement of Teaching in its proposal to 
redefine "scholarship** in ways that emphasize the importance of improving undergraduate 
teaching and learning environments. Literature reviews by Purde (1982) and Aleamoni (1987) 
reveal that considerable evidence supports the use of student ratings as one data source in the 
assessment of course characteristics arid the quality of instruction, particularly when such 
information is used for formative evaluation purposes. While some learning environment 
instruments have been developed for use in higher education settings, (eg., the College and 
University Classroom Environment Inventory fCUCED> Eraser, Trcagust, & Dennis* 1986), no 
measures are known that attempt to assess students^ perceptions in higher education settings of 
the extent to which teaching and learning environment variables enhance learning, particularly 
n newer, important areas such as the development of higher order thinking skills. Eraser, 
Tteagustt Williamson & Tobin (1987), citing die strong tradition of classroom environment 
research using student perceptions measures at the primary and secondary school levels, suggest 
that this research should form a basis for the development of classroom learning environment 
perceptions measures suitable for use in higher education contexts. Manh & Dunkin (1S^2) and 
Mengcs (1992) also recomnoend that researchers in teaching in higher education contexts look 
to die more advanced research base of effective tithing in elementary and secondary schools 
as a starting point in investigating teaching and learning in higher education classes. 
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During the last ten to fiftcrn years, much has been learned about irnportant teaching 
behavioxs and their contribudon to student engagement and the enhancement of student learning 
(Brophy, 1986; Porter & Brophy, 1988) and teaching and learning scholars are moving forward 
with additional models of effective teaching based upon explications and understandings of 
teacher knowledge (Shubnan, 1986; 1987) and teacher expertise (Berliner, 1986; 1989). Alio, 
recent research on the study of learning environments has revealed that student outcomes in 
various educational settings depend to a large extent on the nature of the actual classnxnn 
environment (Raser & Fisher, 1983; Fraser et al., 1987; Marcelo, 1990). Logan and EUett (1988) 
suggest that even though higher education settings are somewhat different teaching and learning 
contexts than public school classrooms, it seems leasonable that the recent process product- 
research literature, literature on classroom learning environments and research on student 
learning, should provide an initial focus for the development of an instrument to measure 
effective teaching and learning environments in higher education contexts. In addition, the extent 
to which students* learning is enhanced in higher education classroom contexts is an important 
assessment concern, from both fonnative and summative evaluation penpectives. This call for 
more comprehensive views of assessment in higher education settings is at odds with the more 
traditional and more narrow use of student ratings of courses alone. 

Recent meu analyses (Menges & Brinko, 1986) and experimental studies of die effecu 
of systematic feedback also note (Marsh & Roche, 1993) reveal that feedback from student 
ratings alone produces small improvements in teaching. However, when this feedback is 
accompanied by professional consultation, the effects on teaching inqsrovement are gready 
enhanced Menges and Brinko (1986) also note that among studies that include consulution, 
there is considerable variation in effect sizes associated with systematic consultation procedures. 
Brinko (1991) con^ares five models of interaction which are applicable to the consultation 
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process. Each imxlel posits a diffeiem rote for the consul^^ For 
exa2iq>tet a consultant's rote might be one of an expert, a facilitator of change, a diagnostician, 
a psychological counselor, or a chaltenger. Participants may assume roles such as a seeker of 
personal growth, a receiver of information, a content expert, and the like in interactions during 
the consulting process. Thus, selecting an appropriate consultation/feedback model to 
tccoounodate individual differences among faculty in higher education settings is an inqxntant 
concern to enhance teaching and teaming environments for students. 

PURPOSE 

The purpose of this paper is to report the results of three years of research and 
development, piloting, and extended field testing of a classroom-based assessment and 
professional consultation system used to assess important teaching and learning variables in 
higher education contexts. Of particular interest is the focus on the total system on enhancing 
learning and newer assessment concerns (e.g., teaching for development of higher order thinking 
skills). The assessment consultation system was developed and piloted as: 1) an alternative to 
evaluation instruments traditionally used in higher education classrooms that are designed to 
measure more narrowly focused instructor or course evaluation characteristics; 2) a system 
reasonably grounded in the emerging research literatures reflecting linkages between etemenu 
of effective teaching and student learning; and 3) an assessment firameworic to tap student 
perceptions in a higher education context of key etements reflected in a classroom-based, on-the- 
job observation system originally designed to evaluate teaching and learning in public school 
classrooms in Louisiana (Ellett, 1990; Ellett, Loup, & Chauvin, 1991a; 1991b). 

METHODOLOGY 

This study was completed in three phases of R&D activity as follows: Phase I: 
Asseiiment instrument and consultation/feedback model development; Phase 11; Initial piloting 
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of the system to examine utility/feasibility for use in undeigraduaie classrooms; Phase QI: 
Studies of validity and reliability of instrumentation; Wmc IV: Piloting of a comprehensive, 
collaboFBtivc/feedback consultation model with higher education faculty. Each of these phases 
is described in the following sections. 

Phase I: instniniftnt and Model Development 
Assessment, Consultation, and Feedback Model 
A comprehensive, classroom-based assessment, consultation and feedback model was 
designed that was grounded in the litcratuxes on teaching and learning cited above (Evans & 
EUett, 1990) and a collaborative process and consultation model (Brinko, 1991). The model was 
enhanced by the use of a combination of thiee triangulated data somces to obtain classrcxnn- 
based assessment information, to be used for fonnative evaluation purposes: 1) student 
perceptions; 2) observations of external observers; and 3) faculty self-assessments. Assessment 
data from these thitc sources are used to identify targets for consultation and discussion in 
fonnative evaluation/professional development conferences with faculty. 

Instrumentation 

Initially^ some 30 different instruments targeting student evaluations of instruction and 
courses in higher education classes were reviewed as to content and response format None of 
these instruments was found to have a primary focus on the actual processes and interactions of 
teaching and learning within the higher education classroom learning environment context With 
few exceptions, these instruments focused upon the evaluation of course content, activities, 
materials and/or personal characteristics of the instructor. Moreover, few of the instruments 
reviewed appeared to be useful for the purposes of providing feedback for formative evaluation 



and instnictioDal improvement Based upon this review and the need for collecting assessment 
data to be used for fanmdve evaluation purposes, a student percepticms assessment instrament 
for use in higher education learning envixonments was developed for this study. Instmment 
content and assessment formats were derived finom the following sources: 1) a synthesis of higher 
education course evaluation and student perceptions measures literatmes; 2) existing noodels and 
perceptions measures in the study of classroom learning environments (e.g.. Moos, 1974; Fiaser, 
Anderson & Walbeig, 1987; leaser & Fisher, 1983; Eraser et aL, 1987, EUett, Loup & Chauvin, 
1991; Loup, Ellett & Chauvin, 1992); 3) a syntheses of process-product research findings 
(Brophy, 1986; Porter & Brophy, 1988); 4) a review and synthesis of large-scale teacher 
perfomiance assessment instruments developed to evaluate teaching in public school contexts in 
Georgia, Florida, South Carolina, North Carolina, Tennessee, and Texas (EUett, Garland & 
Logan, 1987); and 5) more recent assessment concerns such as the caching of thinking skills and 
learning equity reflected in the System for Teaching and learning Assessment and Review 
(STAR) (Ellett, Loup & Chauvin, 1991b). The resulting 68-item draft of a student perceptions 
measure termed the Student Assessment of Teaching and Learning (S ATL)(Evans & EUett, 1992) 
was designed to assess key elements of the interactive nature of classroom teaching and learning 
environments in higher education settings. 

The initial draft of the 68 assessment items of the SATL consisted of indicators of 
effective teaching and learning grouped by four instrument categories: 

1. Preparation and Classroom Management (11 indicators) 

n. Interpersonal Skills (10 indicators) 

UL Enhancement of Learning (24 indicators) 



IV. Student Evaluation Practices (10 indicasors) 
Assessment indicators defining various Teaching and Learning Coaqx»ents of the STAR (EUett, 
et. al, 1991b) weie used to develop the initial pool of assessment indicators for the SATLfor the 
fint ttuee categories. The STAR assessment indicators weie slightly lewonkd wheit appropriate 
to make them more applicable to assessments in higher education contexts. For exaflq>Ie, the 
original STAR indicator sta^emenu Jnthusiasm about teaching, learning and the subject taueht 
is communicated throughout the lcsson. ..was rewritten in the SATL as a set of two assessment 
iiulicator statements... The professor is enthusiastic about teaching this course. ..and.,, The professor 
is enthusiastic about students' le arnji^g course . The SATL assessment indicators 

collectively reflect a complex set of classroom activities, conditions, events and/or behaviors that 
define core elements of teaching and learning rather than specific presciqitive teaching behaviors 
or targeted course characteristics. A list of the SATL assessment indicators classified by each 
of four Assessment Domains can be found in Table 1.0 

The response format for each SAIL assessment indicator is a three-point, forced choice 
scale ruiging from: 1 » INEFFECTIVE (does not positively enhance student learning); 2 « 
EFFECTIVE tfor the most part positively enhances student learning; and 3 « HIGHLY 
EFFECTIVE (consistently enhances student learning to a high degree) . Ihe response fomiat 
represenu a considerable change firom typical course evaluation rating scales (e.g.. Never, 
Sometimes, Always.«.or... Needs Improvement, Satisfactory, Excellent). In assessing each SATL 
indicator, studenu are asked to reflect on the extent to which the indicator as evidenced in the 
class/course contributed to the enhancement of their individual learning. Thus, the student 
response set for assessing each of the SATL indicators is leaming-centered and outcomes-based 
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In addition, for purposes of the pilot study, students were also asked to respond to ten 
items that reflected the degree to which each of ten types of learning (e.g., cndcal analysis and/or 
pioblem solving, creative thinking, learning factual infonnation, etc.) was emphasized in the 
particular course being evaluated In a third pan of tbc SATL, students were also asked to 
provide global, summary evaluation judgements of: 1) the quality of teaching; 2) the contribution 
of the particular course to their learning; and 3) a predicted final course grade. 

Phase n: Initial PUoting 

The SATL and the consultaticm model was initially piloted to develop six case studies of 
teaching and learning in classes taught by undergraduate teaching assistants (Evans & EUett, 
1990). The pilot of the assessment model included collection of student and faculty self- 
perceptions of teaching and learning using the SAIL. Systematic classroom observations were 
completed by external observers who had been trained in the used of the System for Teaching 
and learning Assessment and Review (STAR). Syntheses of SAIL and STAR data were used 
as a basis for ensuing and ongoing consultations with the teaching assistants to inqnove teaching 
and learning in their classes. 

Phase m: Studies of the Validitv and Reliabilitv of the SATL 

San^le 

During the spring of 1992, the SATL was administered to 34 Evening School/C^ontinuing 
Education, off-campus courses (classes) offered during the spring semester, 1992 through 
Louisiana State University (LSU). These classes were located in 12 different education centers 
distributed throughout Louisiana. A wide range of subject areas wu represented in these courses, 
including classes drawn from the biological and social sciences, math, English, foreign languages. 
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educatioa ind foundations classes (e.g., philosophy). The acadcpaic level of courses ranged fitom 
freshman year courses chrough doctoral-level q)ecial topics courses. Class sizes varied fitom nine 
to 55 students. 

Data CoUecuuQ Procedures 
The initial field trial of the SATL rqxKtedbeie was completed during the spring of 1992 
with a random sample of students in Evening School classes on the LSU main canq>us using a 
cross-section of courses representing a wide variety of higher education contexts. One purpose 
of this initial field trial was to establish estimates of itm and instrument category descriptive 
statistics and internal consistency reliabilities. A second putpose was to generate student 
perceptions data for examining construct validity characteristics of the SATL using a series of 
exploratory factor analyses. The SATL was confidentially administered to students in whole 
class groups and required appioximaiely lS-20 minutes to complete. 

Classes were randomly selected within various subject area categories (e.g., social 
sciences* biological sciences, humanities, English^ etc.). Instructors/faculty in the sanq>Ie were 
contacted by letter and asked to cooperate in requesting that their students conqilete the SAIL. 
Conviction of the SATL was described to students as an initial attempt to develop a more 
systematic procedure for evaluating students* perceptions of Evening School courses. It was 
explained that the focus of the study was on students* collective perspectives of Evening School 
courses and that the SATL data would be used to provide meaningful feedback to instructors for 
course improvement purposes. Subsequent class sununaries of SATL data were ccmsidered to 
be confidential and these were not reported to various academic departments. 

SATL data for the snidy were collected during the last two weeks of regularly scheduled 
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Evening School classes during the spring of 1992. Students weic asked to complete the SAIL 
insmiment, a demographic sheet, and a cooment/iesponse form which solicited additional student 
perspectives about the instnictoi/courset the claii^ of directions and items, appropriateness of 
items for the particular course, and any inqxxtant aspects of teaching and learning that were not 
included in the SAIL instrument, etc. Students woe also asked to indicate the length of timi 
required to cmoplete all data collection tasks. 

The SAIL student perception instruments woe distributed and returned by canqpus mail 
and U.S. maiL FoUow-up calls weie made to instnictors after the end of the semester. Of the 
34 classes selected in the original saccule, 28 (82%) participated in the study. The majority of 
student participants were female (67^%) and over the age of 25 (66.8%). Only 74% of students 
in the sample wexe of the traditional college age (18-21). Slightly more than one*half were 
enrolled in a degree progrant Almost two-thirds were part-time students, and 47% had not taken 
a college course within one to three years of the current (spring, 1992) semester. For more than 
22% of the students in the sample, the course evaluated was the fint college course taken in 10 
or more years. 

Data Analyses 

Data were available for various analyses for 448 students distributed across 28 different 
classes. A variety of data analyses were completed to examine the initial psychometric 
characteristics of the SATL. These included: 

1. Descriptive statistical summaries fcir each item (assessment indicator) 

2. A variety of principal component, oblique and orthogonal factor analyses 

3. Alpha reliability analyses for each SATL Domam (Subscale) 
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4. Interconelatioiis among SATL Domains (Subscales) 

5. Intdconelations among expected course grades and SATL Domains (Subscales) 

Results 

A variety of exploratory &ctor analyses were coixq>leted to examine the original content 
classification of SATL items into the four Assessooent Domains. R^ults of a one-factor, 
principal cooqponents solution indicated that the 55 items conqmsing SATL Assessment Domains 
I, n and m meaningfully loaded on a single factor (range of loadings « J8 to .83), with 46 of 
55 loadings exceeding .60. Table 1 summarizes the results of this analysis. Item/factor pattern 
loadings shown in the Table are conelation coefficientt. This one-factor solution accounted for 
4S% of the total variation in the data. 

Subsequently, a series of oblique and orthograal solutions extracting from two to ten 
factors was conq)leted. For oblique solutions, a large number of items cross-loaded on mote than 
one factor. Results of a four-factor, orthogonal solution best depicted the aligmnent of SATL 
indicators with the various Assessment Domains. Factor loadings for this solution ranged fiom 
36 to .77 with 21 loadings at or exceeding .60. Results of this analysis arc shown in Table 2. 
This solution accounted for 56.23% of &e total variance in the data, with 12 of 55 indicators 
(items) cross-loading on more than one factor. Examination of cross-loadings suggested that 
mote generic concepts related to student^teacher communications (eg. clarity in directions, 
explanations and grading) tended to conceptually align with mote than one SATL assessment 
domain. For the most part, the patterning of loadings confirmed the original logical classification 
of SATL indicators by Assessment Domains and represented the best statistical and conceptual 
fit witii the original smicture of the SATL. 
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Descriptive statistics for items and factored dimensions (Domains/Subscales) indicated that 
instructors received relatively consistent scores (80.7% to 86.4% of the maximum possible score) 
across SAIL Assessment Donuuns (Table 3). The lowest percentages were noted for the 
Assessment Domain, Enhancement of Leaming, which includes newer usessment indicators 
targeting the extent to which teaching methods and learning tasks actively involve studenu in the 
development of higher order thinking skills. Interestingly and in^ortantly » assessment scores 
for the various S ATL assessment indicators were lower for indicaton targeting the teaching and 
learning of higher order thinking skills than for other assessment indicators. When viewed from 
a mastery perspective, for example, the percentage of the maximum possible score for all of the 
SATL thinking skills indicators considered collectively was only 17 percent. 

Cronbach Alpha reliability coefficients for each SATL Assessment Domain were as 
follows: I) Preparation and Classroom Management >.89; 2) Interpersonal Skills ».91; 3) 
Enhancement of Learning « .96; and 4) Student Evaluation Practices s.93. 

Table 4 summarizes Pearson product moment correlation coefficients among die various 
SAIL subscales (Assessment Domains) and studentt* summaiy judgements of types of learning 
•tnpliaslsad in the classroom and overall class/course evaluation for the total sample (n«339) of 
iiiukiHi. liiiartimelttiiuns aimmg the various SATL subscales ranged from .76 (p<.0001) 
lluihsiii^MMiiii til Uiiiilng and Inlerperiunal Skills) to .20 (pcOOOl) (Student Evaluation 
^imiliiPi ml Overall llvaluiUon). intercorrelations between the SATL Overall Course 
livituaiUMi tiHii|Hment aiul tiie four SAIL Assessment Domains (Subscales) rangr ' ^m .20 
INHII ) 10 (p<.0001). IntercorrelaUons were also computed between studenu* ^^eptions 
i>l aR|iiL]|«d giades at the beginning uui conclui ion of the coune. Interestingly, these were 
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typically negative in direction, relatively moderate in magnitude, and not statiitically significant 
Phase IV: Piloting of the Comprehensive. Collaborative/Feedback Consultation Model 
During the 1992-1993 academic year, the SATL student and a parallel faculty-self 
perceptions measures were further field tested with a laiiger sample of evening school classes. 
Responses were received from approximately 1,000 students and 80 faculty from a cross section 
of courses, again representing a wide variety of educational contexts. This larger SATL data set 
was subsequently used to further examine construct validity and reliability characteristics. 
Results of subsequent factor analytic worlc confirmed initial clauification of items by subscales 
as reported using dau in the initial pilot of the SATL. 

In addition, participating faculty were sent assessment feedback profiles (Appendix A) 
which includ&d percentages of students' responses for each point on the rating scale for each item 
on the SATL. A selected sample of faculty were also sent Course Evaluation Reports (Appendix 
B) which included a series of histograms depicting various comparisons of usessment data as 
follows: 1) Comparisons of total faculty ratings with total student ratings; 2) Comparisons of 
individual faculty ratings with average faculty ratings; 3) Comparisons of suident ratings for 
individual classes with average student ratings across classes. 

In addition to this comprehensive feedback, twelve classnooms were usessed by external 
observers with the STAR and additional piloting of the consultation model wu Implemented 
The sample for this extended field test activity included volunteer faculty (n*?) and graduate 
teaching assistanu and/or instructors (n»S) from a variety of content areas and settings (e.g., 
nutrition, chemistry lab settings, English, religion, philosophy, etc.)e Each of these twelve 
participanu were observed in the classroom and usessments of teaching and learning were made 
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by trained observers using the STAR. Each also participated in an assessment feedback 
consultation conference in which the observer assuined the role of change facilitator and the 
participant typically seemed to assume the role of content expert. Results weit somewhat mixed 
as receptivity to feedback was varied and most participanu seemed to be laif ely concerned with 
presentation of content rather than with the interaction of elements of content and pedagogy 
which have implications for student understanding and learning through engagement in higher 
order thinking processes. Resulu provided a buis for further development of a consultation 
feedback model which includes encouragement and engagement of participants in conversations 
about teaching and learning, collaborative leflectioni die use of multiple data sources radier 
student perceptions data alone, and* finally, emphasis on teaching as it enhances learning for 
students rather than simply on delivery of content. 

DISCUSSION 

This study describes four phases of R&D activity occurring over a period of tiute years 
to develop and pilot a comprehensive, learning-based assessn^ent and consultation system in 
higher education settings. Itii|Hirtant, and central to tiiis system is Uie emphuls placed on 
classroom activities and events eisenUal to tite enhancement of teaching and learning. The resulu 
of Uiis study are itnpoitant from four mi^ur penpectives. First, Uiey provide empirical support 
for Uie construct validity and reliability of a new, comprehensive measure of sludenu* perceptions 
of characteristics of teaching and learning environments in higher education settings, particularly 
In classes composed of non^traditional adult learners (typically part time and older studenU). 
Results of die factor analyses completed docutnented reasonable alignment of tita SAllr 
assesstnent iiuiicators wlUi the Assessment Domains in which they had been originally higlcally 
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content classified Initial reliability (internal consistency) estimates for the various Assessment 
Donudns are encouraging, though future investigations of stability characteristics of the SATL 
should be (and are presently being) explored 

The focus of the SATL on characteristics of teaching and the enhancement of learning 
makes it rather unique in the highn education learning environment assessment literature. While 
other student perceptions measures of learning environment characteristics have been developed 
for use in higher education settings (e.g., Fraser, Treagust, & Dennis, 19B6), these do not have 
as a central focus studenu* views of the extent to which a set of reasonably research-based 
indicators of teaching and leamlng activities and learning environment variables enhance student 
learning. The SATL is thought to be unique in this regard and in iu focus on newer, important 
assessment concerns such as the teaching and learning of higher order thinking skills u well. 
From both the formative and summative evaluation of teaching perspectives, tite SATL provides 
a rich alternative to tiie wide variety of student course evaluation forms currentiy being used in 
higher education settings that typically have a simpler and more narrow rating focus on initnictcM' 
or course characteriirics alone. In tius regard, tiie SATL represenu a logical extension of newer 
ideas derived bom recent developments of clusroom-based uscssments of teaching and learning 
for public school teachers (EUett, 1990). 

Secondly, Uie results suggest tiut studenu* perceptions of the teaching and learning 
environment as measured by Uie SATL have Uttie or no itlationship to students' beliefs about 
expected course grades at eiUier Uie beginning or die end (after final examinations) of a coune. 
This finding suggesu tiiat Uie SATL response format is relatively independent of such student 
grade expecutions and calls into question past criticisms of Uie use of student perception 
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measures to assess (evaluate) higher education classrooms. Thus, such student perceptions may 
be relatively free of the influence of halQ and tiitchfork effects associated with grades that are 
commonly believed to influence students' perceptions of the quality of teaching in higher 
education classrooms. In addition, smdents* perceptions of the effectiveness of various SATL 
aiieisment indicators in these classes showed only very modest, positive relationships to dieir 
global, summary course evaluation judgments (r=.20 to .29). This finding seems to lend 
additional suppon to the relative independence of the SATL indicator assessments from pitchfork 
and halo effects in evaluation. 

Third, the results showed that the SATL assessment indicators targeting the teaching, 
development and learning of higher order tiiinldng skills received the lowest scores of all SATL 
assessment indicators. Indeed, scores for these indicators were somewhat lower than similarly 
noted low scores obtained tiirough direct classroom-based observations of teaching and learning 
recentiy completed in some 6,000 public school classrooms wdth the STAR (EUett, Loup, & 
Cbauvin, 1990) tim)ughout Louisiana (Hill, Qaudct, Loup, Chauvin, & Naik, 1991). Hiese 
fmdings suggest tiiat these higher education learning environments were not characterized by the 
teaching and learning of higher order cognitive skills. Instead, tiiey were perceived by smdents 
as primarily focused on die acquisition of factual knowledge using traditionil, receptive teaching 
and learning modes. To die extent diat tiiis finding is generalizable to oUier higher education 
institutions and settings, rather serious issues and concerns about the quality of teaching and 
learning in higher education are raised. Among these is the extent to which faculty in such 
settings may need to learn more effective teaching meUiodologiei designed to develop higher 
order thinking among students. 



17 



16 

Fourth, and firom a more personal and clinical perspective, implementation of a 
collaborative feedback/consultation model in this study provided considerable insights about an 
important set of variables that need to be considered in efforts to improve teaching and learning 
in higher education settings. Results of conferences witii faculty and graduate teaching assistants 
served to identify a variety of metaphon that seem to pervade perceptions of teaching and 
learning in higher education classrooms. For example, many faculty viewed themselves as 
essentially "content deliverers" and viewed teaching and learning as essentially moving from "me 
to thee." Thus, tiie concerns reflected in feedback consultations typically focused on how faculty 
might improve tiieir teaching techniques...rather than student learning. Faculty perceptions of the 
teaching role as "fiEicilitator" or "enhancer'' of learning were most evident in conversations witii 
one faculty in a helping professions setting (nurse). 

Working witii faculty to change these habituated perspectives about learning in higher 
education settings raises an important set of concerns. A key to such changes may be findings 
means to increase the legitimacy and perceived importance of Information derived from both 
student evaluation data and direct observation dau. In tiic study reported here^ faculty members 
varied considerably in the extent to which they believed in and valued evaluation results. Some 
became very concerned about evulent low scores (particularly those from Uie SATL), while oUusrs 
became somewhat defensive about sharing results and appeared very hesitant to discuss 
alternative strategies and techniques targeting improvementt. In this regard, graduate teaching 
assistants seenoed more interested in, and valued tiie evaluation results more as a tneans to 
improve, than did regular faculty memben. 

Anotiier concern arising from feedback conferences with faculty relates to self reflections 
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about teaching and learning* These faculty seemed to give priority to a personal focus on 
teaching...and gave little consideration to either improving student leaming.*.or to establishing 
better linkages between the quality of teaching and student learning. They seemed to view and 
deflne their effectiveness as teachers in isolation of concerns for student learning. The qualitadve 
findings reported here strongly suggest that faculty in higher education settings might well profit 
from inservice education that focuses on linkages between effective teaching and student 
learning. The SATL data in this study, for example, strongly suggested that students did not 
believe that faculty and graduate teaching assistanu had any knowledge of student levels of 
misunderstanding of content and subject matter. Developing such understandings, of course, may 
directly d&pend upon the ability to use teaching strategies that target informal aisessmenu of 
student learning during classroom teaching and learning processes. 

The assessment system and consultation model provides alternatives to more traditionally 
used student perceptions instrumenu that focus on course and professor evaluation u opposed 
to actual teaching effectiveness as it relates to student learning . The complete assessment system 
includes measures that triangulate the perspecdves of snidents, external observers, and faculty 
aelf-assessmenu. The SATL instrument used to collect student perceptions data is grounded in 
and derived from a comprehensive, clusroom-based observation system (STAR) originally 
designed to assess die quality of teaching and learning in clusrooms of public school teacben 
for the purpose of professional certification and Improvement (Ellett, Loup, & Chauvin, 1990). 
Interestingly, die resulu from die extended fleld test in Phases III and IV of die smdy suggest 
that college/university classrooms share some common strengths and weaknesses pertaining to 
effective leaching and learning reflected in statewide asaessmenu of public school claaarooms 
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(HiU, Oaudet, Loup, Chauvin» & Naik, 1991). 

In addition, pirliminary findings from piloting of the consultation feedback nxxlel with 
twelve univenity faculty and assistants and earlier case studies with graduate assistants suggest 
that triangulated data sources in the assessment system are a viable means of providing 
meaningful feedback to college/university facul^ for in^ving teaching and learning in higher 
education classrooms. 

The findings from this three-year research and development project to improve teaching 
and learning in higher education settings shed some light on the tiadidonal rhetoric associated 
with the difficulty in developing and conducting valid and reliable assessments in higher 
education clusrooms. In addition, the responK of faculty partidpanu in this study to this 
collaborative consultation model suggests that the traditional rhetoric surrounding the importance 
of matching an observer/assessor's counc/content knowledge with that of the faculty member is 
somewhat suspect Botii die SATL and the STAR appear to usess a variety of important 
elements associated with effective teaching and learning environments (process variables) that 
art generic across different content areu in higher education settings. 

Finally, the results have inqilicationi for using die SATL in die future study of learning 
environment characteristics and for the development of systematic lines of inquiry relating 
teaching and learning environment characteristics to student outcomes. Written comments 
solicited from studenu and instructors in this study indicated that the SATL is easy to admlniiier, 
clear in content and focus, and adequately addresses important upects of a variety of course and 
instructor variables defining multiple classroom learning contexts in higher education. The SATL 
seems efficient u a dau collection device since it requires approximately 20 minutes to 
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administer. Additionally, with its response focus on the extent to which elemenu of teaching and 
classroom activities enhance learning, rather than on broader notions pertaining to piychosocial 
characteristics of the learning environment, the SAIL might be used ai a proxy student 
perceptions measure of student learning and achievement in future research on learning 
environment characteristics in higher education settings. 
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Tabic 1 

Summary of SATL Indicators of Effective Teaching and Learning 
Factor Pattern Loadings for a One-Factor Solution 
for Each Assessment Domain 



INDICATOR NUMBER FACTOR PATTERN LOADING 



ASSESSMENT DOMAIN I: Preparation and Classroom Management 

1. Objectives for the course are clearly conununicated. .63 

2. Objectives for each class arc clearly communicated. .65 

3. Student responsibilities and expecutions are clearly explained .62 

4. The professor is well-prepared for class. .58 

5. Class starts and ends on time. .38 

6. Time during class is efficiently used for teaching and learning. .60 

7. The course activities over the semester are well-organized .64 

8. Class activities help to achieve the objectives of the course. .68 

9. Assigned readings (textbook and/or other) are meaningful. .57 

10. Outside assignments help to achieve the coune objectives and are .57 
integrated with class activities. 

11. Teaching and learning techniques motivate studenu to Itam. .79 
ASSESSMENT DOMAIN U: Inteipersonal Skills 

12. The profesior ii enthuiiaitic about teaching this coune. .59 



or 



Table 1 (Cent) 



13. 


The professor is enthusiastic about smd'nts* leaminE in this 


.67 




course. 




14. 


The professor maintains a climate conducive to learning. 


.69 


15. 


The professor relates the subject to everyday life. 


.65 


16. 


Students are encouraged to express their own ideas. 


.59 


17. 


Students are encouraged to participate in discussions. 


.52 


18. 


A climate of mutual courtesy and respect is maintitined. 


.69 


19. 


The professor demonstrates interest in the progress of individual 


.71 




studenu. 




20. 


The professor is willing to provide outside help and guidance. 


.65 


21. 


The professor is sensitive to the needs and feelings of students. 


.70 


ASSESSMENT DOMAIN III: Enhancement of Learning 




22. 


The professor arouses and maintains students* interest in the subject. 


.78 


23. 


Teaching methods stimulate interest in the course. 


.83 


24. 


The professor's speech is audible and easily undentood. 


.63 


25. 


Directions and explanations related to course content are clear. 


.70 


26. 


Thought'provoking questions are asked. 


.67 


27. 


Questions are asked that allow students to compare and contrast ideas. 


.63 


28. 


1*he professor draws students into discussions among themselves. 




29. 


During classes students are encouraged to interact and learn 


.59 




(torn one another. 
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ERIC 
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Ml f i»(»h ^ Miiii)iiiiii#tHl lieloiG iitt»viiig (HI 10 new Ui\^kH Ml 

M liii)MiMMiM lopu K or iilrim luie HUituiimiy^iU Ml lltr «iul iii (liini. .67 

U i\w pMili'KNoi lu'lpN kiiulenii Ui mgdiiliri) IhliHtiiuitoii miU umkiiimmi .72 

i«liiiiOMNlil|ifi aiiuiiig the lopki in ilie couimp . 

< ( IlKtkiili iiiMiiiiiiiil Ik clei&rly explaincil. .73 

M NuuiriiiM tiie rhtouiuKcii to iiiik queitUmi in lImmm. .6() 

VI NlUiknu' quciiUuiiii tut clcirly ttiiiwerDd. .72 

Ui l 40«iiiio|{ Miiiviticii uic iniplriiieiiieil lal lui upptopriiilc puce. .68 

s t \ uuiir iiiiiUMU itt M an uppmpiimis level of Uifflcjuhy. .69 

(N liMIU'uil pmu ol iltc Icsiioii me cinpliHNlml lo liclp n^uUciiu let&ro. .79 

liir pMilvvMui m'oiK in know when itie muiknti tk) noi underilund. .7.1 
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1 Able I (Cont) 

ASSHSSMHNT DOMAIN IV; Student EvaluaUon Pnu:ikes 

46. The baitU for grodini; rtwie clear. 

47. I he basU for Uic coune grade reflects an appropriate balance 
among tents, papers, aKKignnientSt participation* etc. 

48. 1'ests reflect tlic course content. 

49. The teit quekUont are clearly phrased. 

50. Ilie testv are of reaKonable length. 

51. 1'he tcit i|ucsiioni provide a fair chance for itudents to denumMUate 
their knowledge of the subject. 

52. Course asuigniticnti arc graded fairly. 

53. Tests aitd assignincnis are returned in a reasonable length of time. 

54. Sufficient feedback is provided on all graded work. 

55. Students have op|K)rtunitie» to dctenninc their own progress in the 
course. 



Eigen Value = 24.45 

Proportion of Total Variance lixpluiiicd liy t rai iiH NuluUun ^ 44.41% 



TABLE 2 

Summary of SAIl^ IndlciiUtr Vacun Ptttieni UNulliigN h» « l uui l imitH (hlliugimtil (liuluUitii 
for Each Aiieiiiinienl Uoinuln (n-jl39) 



I II III IV 

ASSESSMENT DOMAIN I: 
Prepwtiion and ClaiinNiiii Maiwgeiiieiti 

1. .54 

2. .58 
^. .61 
4. .60 
5' .46 

6. .59 

7. .69 
« .49 .43 
y .36 .46 

10 .51 

11 .52 .51 

ASSI'.SNMI'.NT DOMAIN II 
liiiei|iriNitiml iSklllw 

1^. .57 
IV .68 

l'^ .47 .48 

I.V .51 

III. .77 

17, .r\ 

IH. .Ml 

23 



ruble 2 (l\mi ) 



20. 
21. 

ASSUSSMiZNT DOMAIN III: 
BiiliMiceineiu of Lenrniiiii 



22. 


.47 


.41 


21 


..1.* 




24. 






2!S. 






2(1. 


.52 




27. 


.62 




2K. 


.70 




2'). 


.69 




Ml 


.52 




M. 


.60 




M 


.57 




\\ 


.49 


M 


W 




.59 




,XI 


.51 


Mt 


.69 




\f 


.55 






,57 




yt. 


.42 


.4N 


Ml 


.:i9 


.46 


41, 


.6i 




42. 


..59 




4.i 


,61 





.39 
.42 
.53 



.36 



I'Nlile 2 (CViiii.) 



A A 

44, 


fell 




4.1. 


.4H 




ASSIiSSMHNT DOMAIN IV: STUDIJN 1 
liVALUATlON PRACnCliS 






46. 




.59 


Al 




•Do 


AU 
*lo. 




AO 


AO 




«oo 


50. 




.66 


51. 




•69 


52. 




•68 


53. 




•73 


54. 




•71 


55. 




•66 


Proportion of Total Variance in Data 


16.8 


14.10 13.0 



•47 



Set ExpiMined by Etich Factor in the 
4-Piu;tor Sulutkm 

l*r()p<itii(ni or the 1 otul Vnrittnce in the Data 
Set Lixplaiiied by u 4-^ttctur Soluiiun - 36.23% 
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TAUUI 3 

NuiiiiiiHry nC Dcacrlpllvc Stutiitici for Each Subicale of the SA'II^ hour-l^iictor Solution for tlie 
1'oImI Hmu\\k of Stiideius 



MiK. %M* 
Poii. Mix. 



SllhHL'NhS 


N 


Mean 


SD 


Score 


Poia. 


I'iMiiiihig Niiti (*liiRiriNini 
MnitiiMeiiWhl (ID** 


448 


27.8 


5.49 


33 


84.2 


liil«i|iriiii»tiMl NkllU (10) 


445 


25.9 


5.07 


30 


86.4 


lUthttituiiMMl i»t l«umliiti 114) 


445 


58.1 


13.40 


72 


80.7 


HlxHkM llvNluHthtii l'tM:lice« 


435 


25.2 


5.65 


30 


83.9 



(tO) 



* M^i Mmx « NultNitulc Meuit Score/Max. Moiaible Score 
Nuiiilier of lleiiiii hi SuhNcale 



if ^ 

O 
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1 AllLli 4 

Muniiiwry uf IntetvurreUtioni Among SATL Subicalei for the TouJ Sample of Studenu 

SimSCALES PCM IS EL SEP TL OE 

l*ra|iiriili(m iind Claisroom 1.0 .75 .75 .68 .57 .27 
MiiiiKement (PCM) 

lntei|icninn«l Skill! (IS) 1.0 .76 .62 .57 .29 

liiiliHticcntom of Uarning 1.0 .70 .65 .28 
(III.) 

Nliiileiit KvHluutioii 1.0 .46 .20 
Pnivllcett (SUP) 

Types of Uamlng (TL) 1.0 .2.1 

Ovcmll Evaluation (OE) ^ 1.0 

•p<.(K)l (all) 
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APPENDIX A 
Course Evaluation Report Piunies 
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AI'I'liNDIX II 
I'ouiic liviiluMlhm Ke|Mirl lll)<litgiHiiiN 
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